Method and System for Defect Classification

ABSTRACT

Defect classification includes acquiring one or more images of a specimen, receiving a manual classification of one or more training defects based on one or more attributes of the one or more training defects, generating an ensemble learning classifier based on the received manual classification and the attributes of the one or more training defects, generating a confidence threshold for each defect type of the one or more training defects based on a received classification purity requirement, acquiring one or more images including one or more test defects, classifying the one or more test defects with the generated ensemble learning classifier, calculating a confidence level for each of the one or more test defects with the generated ensemble learning classifier and reporting one or more test defects having a confidence level below the generated confidence threshold via the user interface device for manual classification.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is related to and claims benefit of the earliestavailable effective filing date from the following applications. Thepresent application constitutes a divisional patent application of U.S.patent application entitled METHOD AND SYSTEM FOR DEFECT CLASSIFICATION,naming Li He, Chien-Huei Adam Chen, Sankar Venkataraman, John R. JordanIII, Huajun Ying and Harsh Sinha as inventors, filed Jun. 24, 2015,application Ser. No. 14/749,316, which is a regular (non-provisional)patent application of U.S. Provisional Patent Application entitledMETHOD FOR HIGH PURITY DEFECT CLASSIFICATION AND DEFECT DATA ANALYSIS,naming Li He, Chien Huei Adam Chen, Sankar Venkataraman, John R. JordanIII, Huajun Ying and Sinha Harsh as inventors, filed May 8, 2015Application Ser. No. 62/158,605. U.S. patent application Ser. No.14/749,316 and U.S. Provisional Patent Application No. 62/158,605 areincorporated by reference herein in their entirety.

TECHNICAL FIELD

The present invention generally relates to defect review andclassification, and, in particular, to the automatic generation of adefect classifier that provides high purity defect classification.

BACKGROUND

Fabricating semiconductor devices such as logic and memory devicestypically includes processing a substrate such as a semiconductor waferusing a large number of semiconductor fabrication processes to formvarious features and multiple levels of the semiconductor devices. Assemiconductor device size become smaller and smaller, it becomescritical to develop enhanced inspection and review devices andprocedures. One such procedure includes classification and analysis ofdefects on a specimen, such as a wafer. As used throughout the presentdisclosure, the term “wafer” generally refers to substrates formed of asemiconductor or non-semiconductor material. For example, asemiconductor or non-semiconductor material may include, but are notlimited to, monocrystalline silicon, gallium arsenide, and indiumphosphide.

Defect review is a process by which a review tool reviews defectsacquired by an inspector or inspection tool. Defect review requires theclassification of defects and the differentiation, or separation ofdefect types based on a set of calculated defect attributes. However,current defect classification approaches have a number of limitations.

First, decision trees for defect classification are often manuallycreated using calculated attributes, which is a time consuming process.In this case, a user has to select the best attributes for each node ofa decision tree from a large number (e.g., greater than 80) ofattributes. In addition, the tree size may become large (e.g., greaterthan 50 nodes). In addition, the quality of a manually created tree isrelated to a user's interpretation and understanding of the availableattributes and the decision tree creation process. Further, currentapproaches to measure defect type separability are limited. Priorapproaches require a user to find the best attributes for type pairsmanually in order to separate two defect types. Moreover, currentapproaches to classifier monitoring through the production is timeconsuming and complex.

As such, it would be advantageous to provide a system and method thatprovides improved defect classification, defect type separability andclassifier monitoring that cures the defects identified above.

SUMMARY

A method for defect classification is disclosed. In one illustrativeembodiment, the method includes acquiring one or more images of aspecimen, the one or more images including a plurality of defect types.In another illustrative embodiment, the method includes receiving asignal from a user interface device indicative of a manualclassification of one or more training defects of the specimen based onone or more attributes of the one or more training defects. In anotherillustrative embodiment, the method includes generating an ensemblelearning classifier based on the received manual classification and theattributes of the one or more training defects. In another illustrativeembodiment, the method includes generating a confidence threshold foreach defect type of the one or more training defects based on a receivedclassification purity requirement. In another illustrative embodiment,the method includes acquiring one or more images including one or moretest defects. In another illustrative embodiment, the method includesclassifying the one or more test defects with the generated ensemblelearning classifier. In another illustrative embodiment, the methodincludes calculating a confidence level for each of the one or more testdefects with the generated ensemble learning classifier. In anotherillustrative embodiment, the method includes reporting one or more testdefects having a confidence level below the generated confidencethreshold via the user interface device for manual classification.

A method for determining a defect type pair score is disclosed. In oneillustrative embodiment, the method includes acquiring one or moreimages including a plurality of defect types, the plurality of defecttypes including a first defect type and at least a second defect type.In another illustrative embodiment, the method includes generating afirst ensemble learning classifier for the first defect type and the atleast a second defect type. In another illustrative embodiment, themethod includes calculating, with the first ensemble learningclassifier, a mean decrease in an accuracy index for each of a pluralityof attributes associated with the first defect type and the at least asecond defect type. In another illustrative embodiment, the methodincludes identifying a selected number of attributes having the largestmean decrease in accuracy index. In another illustrative embodiment, themethod includes generating a second ensemble learning classifier withthe identified selected number of attributes having the largest meandecrease in accuracy index. In another illustrative embodiment, themethod includes determining a training error associated with the secondgenerated ensemble learning classifier. In another illustrativeembodiment, the method includes calculating a defect type pair scoreassociated with the first defect type and the second defect type basedon the determined training error.

A method for determining the sufficiency of defect data forclassification is disclosed. In one illustrative embodiment, the methodincludes acquiring a set of defect data from a specimen, the defect dataincluding imagery data associated with a plurality of defects includinga plurality of defect types. In another illustrative embodiment, themethod includes receiving a signal from a user interface deviceindicative of a manual classification of the plurality of defects. Inanother illustrative embodiment, the method includes distributing defectdata of at least the first defect type into N groups of data. In anotherillustrative embodiment, the method includes identifying a group of theN groups of data as containing test data. In another illustrativeembodiment, the method includes identifying N-1 groups of data of thedistributed defect data not identified as containing test data ascontaining training data. In another illustrative embodiment, the methodincludes, for at least a first group of the N groups, incrementallygenerating a series of classifiers based on the training defect datacontained in the N-1 groups of data, wherein each classifier isgenerated with an incremented percentage of at least a first defect typecontained within the training defect data of the N-1 groups of data. Inanother illustrative embodiment, the method includes determining anaccuracy value for each of the series of classifiers for at least thefirst defect type by applying each of the series of classifiers to thetest data not contained in the N-1 groups of the distributed defectdata. In another illustrative embodiment, the method includes generatinga defect data sufficiency score, for at least the first defect type,based on a generated accuracy score for at least the first group of Ngroups and at least one additional generated accuracy score for at leastone additional group of the N groups.

An apparatus for defect classification is disclosed. In one illustrativeembodiment, the apparatus includes an inspection tool. In anotherillustrative embodiment, the inspection tool includes one or moredetectors configured to acquire one or more images of at least a portionof a specimen. In another illustrative embodiment, the apparatusincludes a user interface device. In another illustrative embodiment,the apparatus includes a controller. In another illustrative embodiment,the controller includes one or more processors communicatively coupledto the one or more detectors of the inspection tool, wherein the one ormore processors are configured to execute a set of program instructionsstored in memory. In another illustrative embodiment, the set of programinstructions are configured to cause the one or more processors to:receive the one or more images from the one or more detectors of theinspection tool; receive a signal from a user interface deviceindicative of a manual classification of one or more training defects ofthe specimen based on one or more attributes of the one or more trainingdefects; generate an ensemble learning classifier based on the receivedmanual classification and the attributes of the one or more trainingdefects; generate a confidence threshold for each defect type of the oneor more training defects based on a received classification purityrequirement; acquire one or more images including one or more testdefects; classify the one or more test defects with the generatedensemble learning classifier; calculate a confidence level for each ofthe one or more test defects with the generated ensemble learningclassifier; and report one or more test defects having a confidencelevel below the generated confidence threshold via the user interfacedevice for manual classification.

An apparatus for determining one or more defect type pair scores isdisclosed. In one illustrative embodiment, the apparatus includes aninspection tool. In another illustrative embodiment, the inspection toolincludes one or more detectors configured to acquire one or more imagesof at least a portion of a specimen. In another illustrative embodiment,the apparatus includes a user interface device. In another illustrativeembodiment, the apparatus includes a controller. In another illustrativeembodiment, the controller include one or more processorscommunicatively coupled to the one or more detectors of the inspectiontool, wherein the one or more processors are configured to execute a setof program instructions stored in memory. In another illustrativeembodiment, the set of program instructions are configured to cause theone or more processors to: receive the one or more images from theinspection tool, the one or more images including a plurality of defecttypes, the plurality of defect types including a first defect type andat least a second defect type; generate a first ensemble learningclassifier for the first defect type and the at least a second defecttype; calculate, with the first ensemble learning classifier, a meandecrease in an accuracy index for each of a plurality of attributesassociated with the first defect type and the at least a second defecttype; identify a selected number of attributes having the largest meandecrease in accuracy index; generate a second ensemble learningclassifier with the identified selected number of attributes having thelargest mean decrease in accuracy index; determine a training errorassociated with the second generated ensemble learning classifier; andcalculate a defect type pair score associated with the first defect typeand the second defect type based on the determined training error.

An apparatus for determining sufficiency of defect data forclassification is disclosed. In one illustrative embodiment, theapparatus includes an inspection tool. In another illustrativeembodiment, the inspection tool includes one or more detectorsconfigured to acquire one or more images of at least a portion of aspecimen. In another illustrative embodiment, the apparatus includes auser interface device. In another illustrative embodiment, the apparatusincludes a controller. In another illustrative embodiment, thecontroller includes one or more processors communicatively coupled tothe one or more detectors of the inspection tool, wherein the one ormore processors are configured to execute a set of program instructionsstored in memory. In another illustrative embodiment, the set of programinstructions are configured to cause the one or more processors to:receive the set of defect data from a specimen, the defect dataincluding imagery data associated with a plurality of defects includinga plurality of defect types; and generate a defect data sufficiencyscore, for at least a first defect type, based on a generated accuracyscore for at least a first group of N groups of the defect data and atleast one additional generated accuracy score for at least oneadditional group of the N groups of defect data.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not necessarily restrictive of the invention as claimed. Theaccompanying drawings, which are incorporated in and constitute a partof the specification, illustrate embodiments of the invention andtogether with the general description, serve to explain the principlesof the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The numerous advantages of the disclosure may be better understood bythose skilled in the art by reference to the accompanying figures inwhich:

FIG. 1 is a block diagram view of a system for defect classification andanalysis, in accordance with one embodiment of the present disclosure.

FIG. 2 is a flow diagram illustrating steps performed in a method fordefect classification, in accordance with one embodiment of the presentdisclosure.

FIG. 3 is a flow diagram illustrating steps performed in a method fordetermining a defect type pair score, in accordance with one embodimentof the present disclosure.

FIG. 4 is a flow diagram illustrating steps performed in a method for adetermining defect data sufficiency score, in accordance with oneembodiment of the present disclosure.

DETAILED DESCRIPTION OF THE INVENTION

Reference will now be made in detail to the subject matter disclosed,which is illustrated in the accompanying drawings.

Referring generally to FIGS. 1 through 4, a method and system for defectclassification and analysis are described in accordance with the presentdisclosure. Embodiments of the present disclosure are directed to theautomatic classification of defects provided by an inspection or reviewtool. In some embodiments, the present disclosure provides for theautomatic generation and/or training of an ensemble learning baseddefect classifier, which provides high purity output on defectclassification. In addition, the ensemble learning classifier of thepresent disclosure provides for classifier monitoring. Additionalembodiments of the present disclosure provide for defect type pairseparability analysis. In this sense, embodiments of the presentdisclosure are directed to the determination of a defect type pair scorein analyzing whether one or more attributes associated with a pair ofdefects is sufficient to separate the two defects. Additionalembodiments of the present disclosure provide for the determination ofthe sufficiency of a given set of defect data for the purposes ofclassifying the given defect data.

FIG. 1 illustrates a conceptual block diagram view of a system 100 fordefect classification and analysis, in accordance with one or moreembodiments of the present disclosure. In one embodiment, the system 100includes an inspection tool 102. In one embodiment, the inspection tool102 is configured to measure one or more characteristics of one or moredefects disposed on or in the surface of a specimen 112, such as, butnot limited to, a semiconductor wafer (e.g., Si wafer).

The inspect tool 102 may include any inspection configuration known inthe art capable of defect review. In one embodiment, as depicted in FIG.1, the inspection tool 102 is an e-beam defect review (EDR) tool. Forexample, as shown in FIG. 1, the inspection tool 102 includes anelectron source 101 (e.g., electron gun), a detector 103 (e.g.,secondary electron detector) and any one or more electron-opticalcomponents 105 known in the art for carrying out defect review.

It is noted herein that the scope of the present disclosure is notlimited to the EDR configuration of system 100 or electron-beam reviewin general. In one embodiment, the inspection tool 102 may be configuredas a light-based inspection tool. For example, the inspection tool 102may be, but is not required to be, configured for darkfield inspection.By way of another example, the inspection tool 102 may be, but is notrequired to be, configured for brightfield inspection.

In one embodiment, the inspection tool 102 includes one or more lightsources (not shown) to illuminate the specimen 112. The light source mayinclude any light source known in the art. For example, the light sourcemay include a narrow band light source, such as a laser source. By wayof another example, the light source may include a broad band lightsource, such as a discharge lamp or a laser-sustained plasma (LSP) lightsource. In another embodiment, the light source may be configured todirect light to the surface of the specimen 112 (via various opticalcomponents) disposed on the sample stage 115. Further, the variousoptical components of the inspection tool 102 are configured to directlight reflected, scattered, and/or diffracted from the surface of thespecimen 112 to a detector (not shown) of the inspection tool 102. Thedetector may include any appropriate light detector known in the art. Inone embodiment, the detector may include, but is not limited to, acharge coupled device (CCD) detector, a photomultiplier tube (PMT)detector, and the like.

It is noted that for purposes of simplicity the inspection tool 102 hasbeen depicted in a simplified block diagram. This depiction, includingthe components and geometrical configuration, is not limiting and isprovided for illustrative purposes only. It is recognized herein thatthe inspection tool 102 may include any number of optical elements(e.g., lenses, mirrors, filters, beam splitter and the like), energysources (e.g., light source or electron source) and detectors (e.g.,light detector or secondary electron detector) to carry out theinspection of one or more portions of specimen 112 disposed on thesample stage 115.

In another embodiment, the system 100 includes a controller 104. In oneembodiment, the controller 104 is communicatively coupled to theinspection tool 102. For example, the controller 104 may be coupled tothe output of detector 103 of the inspection tool 102. The controller104 may be coupled to the detector 103 in any suitable manner (e.g., byone or more transmission media indicated by the line shown in FIG. 1)such that the controller 104 can receive the output generated by theinspection tool 102.

In one embodiment, the controller 104 includes one or more processors106 communicatively coupled to the detector 103 and memory 108. In oneembodiment, the one or more processors 106 are configured to execute aset of program instructions 116 maintained in memory 108.

The one or more processors 106 of controller 104 may include any one ormore processing elements known in the art. In this sense, the one ormore processors 106 may include any microprocessor-type deviceconfigured to execute software algorithms and/or instructions. In oneembodiment, the one or more processors 106 may consist of a desktopcomputer, mainframe computer system, workstation, image computer,parallel processor, or other computer system (e.g., networked computer)configured to execute a program configured to operate the system 100, asdescribed throughout the present disclosure. It should be recognizedthat the steps described throughout the present disclosure may becarried out by a single computer system or, alternatively, multiplecomputer systems. In general, the term “processor” may be broadlydefined to encompass any device having one or more processing elements,which execute program instructions from a non-transitory memory medium(e.g., memory 108). Moreover, different subsystems of the system 100(e.g., inspection tool 102, display 114, or user interface 110) mayinclude processor or logic elements suitable for carrying out at least aportion of the steps described throughout the present disclosure.Therefore, the above description should not be interpreted as alimitation on the present invention but merely an illustration.

The memory medium 108 may include any storage medium known in the artsuitable for storing program instructions executable by the associatedone or more processors 106. For example, the memory medium 108 mayinclude a non-transitory memory medium. For instance, the memory medium108 may include, but is not limited to, a read-only memory, a randomaccess memory, a magnetic or optical memory device (e.g., disk), amagnetic tape, a solid state drive and the like. In another embodiment,it is noted herein that the memory 108 is configured to store one ormore results from the inspection tool 102 and/or the output of thevarious steps described herein. It is further noted that memory 108 maybe housed in a common controller housing with the one or more processors106. In an alternative embodiment, the memory 108 may be locatedremotely with respect to the physical location of the processors 106 andcontroller 104. For instance, the one or more processors 106 ofcontroller 104 may access a remote memory (e.g., server), accessiblethrough a network (e.g., internet, intranet and the like). In anotherembodiment, the memory medium 108 stores the program instructions 116for causing the one or more processors 106 to carry out the varioussteps described through the present disclosure.

In another embodiment, the controller 104 of the system 100 may beconfigured to receive and/or acquire data or information from othersystems (e.g., inspection results from an inspection system or metrologyresults from a metrology system) by a transmission medium that mayinclude wireline and/or wireless portions. In this manner, thetransmission medium may serve as a data link between the controller 104and other subsystems of the system 100. Moreover, the controller 104 maysend data to external systems via a transmission medium (e.g., networkconnection).

In another embodiment, the system 100 includes a user interface 110. Inone embodiment, the user interface 110 is communicatively coupled to theone or more processors 106 of controller 104. In another embodiment, theuser interface device 110 may be utilized by controller 104 to acceptselections and/or instructions from a user. In some embodiments,described further herein, a display 114 may be used to display data to auser (not shown). In turn, a user may input, via user input device 113,a selection and/or instructions responsive to data displayed to the uservia the display device 114.

The user interface device 110 may include any user interface known inthe art. For example, the user input device 113 of the user interface110 may include, but is not limited to, a keyboard, a keypad, atouchscreen, a lever, a knob, a scroll wheel, a track ball, a switch, adial, a sliding bar, a scroll bar, a slide, a handle, a touch pad, apaddle, a steering wheel, a joystick, a bezel input device or the like.In the case of a touchscreen interface device, those skilled in the artshould recognize that a large number of touchscreen interface devicesmay be suitable for implementation in the present invention. Forinstance, the display device 114 may be integrated with a touchscreeninterface, such as, but not limited to, a capacitive touchscreen, aresistive touchscreen, a surface acoustic based touchscreen, an infraredbased touchscreen, or the like. In a general sense, any touchscreeninterface capable of integration with the display portion of a displaydevice is suitable for implementation in the present invention. Inanother embodiment, the user input device 113 may include, but is notlimited to, a bezel mounted interface.

The display device 114 may include any display device known in the art.In one embodiment, the display device 114 may include, but is notlimited to, a liquid crystal display (LCD). In another embodiment, thedisplay device 114 may include, but is not limited to, an organiclight-emitting diode (OLED) based display. In another embodiment, thedisplay device 114 may include, but is not limited to a CRT display.Those skilled in the art should recognize that a variety of displaydevices may be suitable for implementation in the present invention andthe particular choice of display device 114 may depend on a variety offactors, including, but not limited to, form factor, cost, and the like.In a general sense, any display device capable of integration with auser input device (e.g., touchscreen, bezel mounted interface, keyboard,mouse, trackpad, and the like) is suitable for implementation in thepresent invention.

In one embodiment, the one or more processors 106 of controller 104 areprogrammed to carry out one or more steps of a defect classifiergeneration and defect classification procedure. In one embodiment, theone or more processors 106 may automatically generate an ensemblelearning classifier based on manually classified training defects andcalculated attributes. In addition, the one or more processors 106 mayautomatically calculate a per-type confidence threshold in accordancewith a received user-selected purity requirement. The one or moreprocessors 106 may calculate a confidence measure for each test defect(e.g., test defect of a production wafer). In addition, in production,the one or more processors 106 may report defects having a confidencelevel less than the confidence threshold to the user interface device110 for manual review by a user.

In another embodiment, the one or more processors 106 of controller 104are programmed to carry out one or more steps of a defect type pairscore determination procedure. In one embodiment, the defect type pairscore is calculated, by the one or more processors 106, on the defecttypes of a defect type pair. In this regard, a defect type pair score iscalculated between any two defect types. For instance, in the case wherea set of defect data includes defect types A, B and C, the one or moreprocessors 106 may calculate the following: a defect type pair scorebetween defect type A and B; a defect type pair score between B and C;and a defect type pair score between A and C. In another embodiment, fordefect pair data, an ensemble learning classifier (e.g., random forestclassifier) may be trained and the mean accuracy decrease index for eachattribute may be calculated. After sorting of the mean accuracy decreaseindex, the first N attributes are used to train an additional randomforest classifier, where training error (e.g., out-of-bag (OOB) error)is calculated. A type pair having a score lower than a pre-definedthreshold indicates that the attributes are inadequate for separatingthe two defect types.

In another embodiment, the one or more processors 106 are programmed tocarry out one or more steps of a defect data sufficiency determinationprocedure. In one embodiment, the one or more processors 106 receivemanually classified defect data and attributes, which are distributedinto N groups (e.g., folders). In addition, defect data of the N-1groups (e.g., folders) may be used as training data, with the remaininggroup used as test data. In calculating the score for a particulardefect type, the one or more processors 106 may incrementally increasethe number of defects of the particular defect type in the training data(e.g., incrementally increase from 10% to 100%). In this case, the oneor more processors 106 may build a classifier at each incremental stepand apply it to the test data. An accuracy score is then calculatedbased on the deviation of the accuracy at each incremental defectcontent step. A data sufficiency score is then generated by repeatingthis process for each group (e.g., folder) and averaging the resultsacross all the data groups (e.g., folders). This process may then berepeated for each defect type.

The embodiments of the system 100 illustrated in FIG. 1 may be furtherconfigured as described herein. In addition, the system 100 may beconfigured to perform any other step(s) of any of the methodembodiment(s) described herein.

FIG. 2 is a flow diagram illustrating steps performed in a method 200 ofdefect classification, in accordance with one embodiment of the presentdisclosure. It is noted herein that the steps of method 200 may beimplemented all or in part by the system 100. It is further recognized,however, that the method 200 is not limited to the system 100 in thatadditional or alternative system-level embodiments may carry out all orpart of the steps of method 200.

In step 202, one or more images 107 including multiple defect types areacquired. In one embodiment, as shown in FIG. 1, at least a portion ofthe defects contained in one or more images 107 of a specimen 112 arecapable of serving as training defects. In one embodiment, theinspection tool 102 acquires the one or more images 107 and transmitsthe one or more images 107 to the one or more controllers 104. It isfurther contemplated that the one or more images 107 may be stored inmemory 108 and used for later analysis.

In step 204, a manual classification of the one or more training defectsis carried out. For example, a user may manually classify the trainingdefects contained in the acquired one or more images 107. For instance,a user may manually classify the training defects via user interfacedevice 110 based on one or more attributes of the one or more trainingdefects. In turn, the user interface device 110 may transmit a signalindicative of a manual classification of the one or more trainingdefects of the specimen 112 to the one or more processors 106 ofcontroller 104. In another embodiment, the controller 104 may receivethe manual classification of the training defects and store the resultin memory 108. The one or more attributes used to carry out theclassification of step 204 include any one or more attributes that canbe derived from a defect inspection or review tool. For example, the oneor more attributes may include, but are not limited to, image featureamounts, defect coordinates, composition analysis results, manufactureinitiation history data, or machine QC (Quality Control) data. Further,in some embodiments, the one or more attributes may be obtained frommultiple types of defect inspection tools or systems, such as, but notlimited to, an optical or SEM foreign matter inspection machine, apattern inspection machine, a defect review machine, SPM, or anelemental analysis machine. Attributes suitable for classification ofdefects are described in U.S. Pat. No. 7,602,962, issued on Oct. 13,2009, which is incorporated herein by reference in the entirety.

In one embodiment, the attributes of the one or more training defectsmay be processed by an automatic classification function, such as, butnot limited to, a real time automatic classification (RT-ADC) toclassify one or more training defects. It is noted that the utilizationof RT-ADC provides for a “rough” automatic classification of trainingdefects without sacrificing high processing speeds. Real time automaticclassification is described generally in U.S. Pat. No. 7,602,962, issuedon Oct. 13, 2009, which is incorporated above by reference in theentirety.

In step 206, an ensemble learning classifier is generated. In oneembodiment, the ensemble learning classifier is generated, or trained,based on the manual classification of training defects of step 204 andthe one or more attributes. In one embodiment, the one or moreprocessors 106 of controller 104 may generate, or train, the ensemblelearning classifier and store the ensemble learning classifier in memory108. In one embodiment, the ensemble learning classifier is a randomforest classifier. The one or more processors 106 may train a randomforest classifier that operates by constructing multiple decision treesduring a training period and outputting a class that is the mode of theclasses of the individual trees. In this regard, the one or moreprocessors 106 may use the manual classification of the training defectsand the associated attributes to train a random forest classifier. Theimplementation of a random forest classifier is described generally byBreiman in Random Forests, Machine Learning, Vol. 45, Issue 1, pp. 5-32(2001), which is incorporated herein by reference in the entirety.Random forests are also discussed by Kulkarni et al. in Random ForestClassifiers: A Survey and Future Research Directions, InternationalJournal of Advanced Computing, Vol. 36, Issue 1, pp. 1144-1153 (2013),which is incorporated herein by reference in the entirety.

In another embodiment, the ensemble learning classifier is a supportvector machine (SVM). The one or more processors 106 may use the manualclassification of the training defects and the associated attributes totrain a SVM-based classifier. The implementation of a SVM-basedclassifier is described generally by Xie et al. in Detection andClassification of Defect Patterns in Optical Inspection Using SupportVector Machines, Intelligent Computing Theories Lecture Notes inComputer Science, Vol. 7995, pp. 376-384 (2013), which is incorporatedherein by reference in the entirety.

In step 208, a confidence threshold for each defect type of the one ormore training defects is generated. In one embodiment, the one or moreprocessors 106 generate a confidence threshold for each defect type ofthe one or more training defects based on a confidence thresholdreceived from user interface 110. In another embodiment, the confidencethreshold is generated by one or more processors 106 via across-validation procedure. In another embodiment, the confidencethreshold may be manually set via user input (e.g., user input via userinterface 110).

For example, a user may select a purity requirement, or purity level,via user interface 110. In turn, the user interface device 110 maytransmit a signal indicative of the selected purity requirement to theone or more processors 106 of controller 104. The controller 104 maythen store the selected purity requirement in memory 108. It is notedherein that the received purity requirement may be a function of avariety of parameters. For instance, the received purity requirement maydepend on a user preference and/or the expected defect types present ona given production wafer. For instance, a user may select a singleclassification purity requirement (e.g., 90%) on all defect types. Inanother instance, a user may select a first classification purityrequirement (e.g., 90%) on a first defect type and a secondclassification purity requirement (e.g., 85%) on a second defect typeand so on. It is further noted that in some embodiments a heightenedpurity requirement correlates to a higher confidence threshold. In thisregard, the one or more processors 106 of controller 104 automaticallyadjust the confidence threshold in response to a user input via userinterface 110.

In step 210, one or more images including one or more test defects areacquired. In one embodiment, the one or more test defects are disposedon a different specimen (e.g., wafer) than the one or more trainingdefects. For example, the inspection tool 102 may acquire imagery datafrom a production specimen 112 (e.g., wafer), whereby the controller 104extracts one or more test defects from the given imagery data. Further,the training defects, used to generate the ensemble learning classifier,may be extracted from imagery data from a training specimen 112. Inanother embodiment, the one or more test defects and the one or moretraining defects of the present disclosure are disposed on the samespecimen 112 (e.g., wafer). In this regard, the imagery data (e.g., oneor more images) used to extract the training defects is also used toacquire the one or more test defects of the present disclosure.

In step 212, the one or more test defects are classified with thegenerated ensemble learning classifier. In one embodiment, the one ormore processors 106 may retrieve the ensemble learning classifier storedin memory 108 (see step 206) and apply the ensemble learning classifierto one or more test defects acquired in step 210. In this regard, oncethe ensemble learning classifier has been trained in step 206 it maythen be used to classify one or more test defects contained in imagerydata acquired from a given sample (e.g., production wafer).

In step 214, a classification confidence level is calculated for each ofthe one or more test defects. In one embodiment, the one or moreprocessors 106 calculates a confidence level for each, or at least some,of the one or more test defects with the ensemble learning classifiertrained in step 206. It is noted herein that the confidence level of theclassification for the one or more defects may be calculated in anymanner known in the art. In one embodiment, the one or more processors106 calculate the confidence level of the one or more test defects via avoting procedure. For example, in the case of a random forestclassifier, each tree of the random forest classifier has aclassification output referred to herein as a “vote.” In this case, theconfidence level for the one or more test defects may be, but is notrequired to be, calculated via a major two vote scheme given by:

${Confidence} = {\frac{{Majority}\mspace{14mu} {Vote}}{{Sum}\mspace{14mu} {of}\mspace{14mu} {Major}\mspace{14mu} {Two}\mspace{14mu} {Vote}}\sqrt{\frac{{Sum}\mspace{14mu} {of}\mspace{14mu} {Major}\mspace{14mu} {Two}\mspace{14mu} {Vote}}{{Total}\mspace{14mu} {Vote}}}}$

In step 216, the one or more test defects having a confidence levelbelow the generated confidence threshold are reported via the userinterface device for manual classification. In one embodiment, the oneor more processors 106 compare the calculated confidence level of step214 to the generated confidence threshold of step 208. In the case wherethe calculated confidence level is above the confidence threshold, theclassification is accepted and stored in memory 108. In the case wherethe calculated confidence level is below the confidence threshold, thegiven one or more test defects are indexed and reported to a user formanual classification. For example, in the case where the calculatedconfidence level is below the confidence threshold for one or more testdefects, the imagery data and/or known attributes associated with thegiven one or more test defects may be displayed on display 114. In thisregard, a user may perform a manual classification of these test defectsvia user input device 113. In turn, the user interface device 110 maythen transmit the manual calculation of these test defects and storethem in memory 108. In this regard, the automatically classified testdefects (classified with the ensemble learning classifier) and themanually classified test defects (classified via user input) may then beaggregated into an integrated database for review and/or analysis.

In addition, after the one or more test defects have been analyzed, themethod may move to step 218, whereby the steps 212-216 are repeated ifadditional test defects require analysis. In the case where noadditional test defects require analysis, the process ends.

In another embodiment, a confidence level of each defect type of the oneor more test defects may be used by the one or more processors 106 tomonitor the effectiveness of the ensemble learning classifier generatedby system 100. For example, an average confidence level for each defecttype may be used by the one or more processors 106 to monitor theensemble learning classifier of system 100. In this regard, the averageconfidence level associated with the classification of each defect typemay be used to monitor the ensemble learning classifier of system 100 byindicating the Pareto change of the defect types and/or the creation ofa new defect type in a wafer production process.

FIG. 3 is a flow diagram illustrating steps performed in a method 300 ofdetermining a defect type pair score, in accordance with one embodimentof the present disclosure. It is noted herein that the steps of method300 may be implemented all or in part by the system 100. It is furtherrecognized, however, that the method 300 is not limited to the system100 in that additional or alternative system-level embodiments may carryout all or part of the steps of method 300. It is further noted that theembodiments and examples described in the context of method 200 shouldbe interpreted to extend to method 300 (and vice versa) unless otherwisenoted.

In step 302, one or more images 107 are acquired including multipledefect types. In one embodiment, the multiple defect types include afirst defect type and at least a second defect type. In this regard, thefirst defect type and the second defect type form a defect type pair. Itis noted herein that a given set of inspection data may include a highnumber of defect types. In this regard, any combination of two differentdefect types may form a defect type pair.

In step 304, a first ensemble learning classifier is generated, ortrained, for a pair of defect types. In one embodiment, the one or moreprocessors 106 may train a first ensemble learning classifier for a pairof defect types. For example, the one or more processors 106 maygenerate a first ensemble learning classifier for a first defect type(A) and a second defect type (B). By way of another example, the one ormore processors 106 may generate a first ensemble learning classifierfor a second defect type (B) and a third defect type (C). By way ofanother example, the one or more processors 106 may generate a firstensemble learning classifier for a first defect type (A) and a thirddefect type (C). It is noted herein that the labeling of the defecttypes above is not limiting and is provided merely for illustrativepurposes. Further, the various defect types are not limited to theexamples provided above and may include any pair of defect types presentin the acquired defect data.

The first ensemble learning classifier may include any ensemble learningclassifier known in the art. For example, as discussed previouslyherein, the first ensemble learning classifier may include, but is notlimited to, a random forest classifier or a SVM-based classifier. It isfurther noted that the one or more processors 106 of controller 104 maytrain the ensemble learning classifier (e.g. random forest classifier)by randomly selecting a subset of the training data and then randomlyselecting a subset of the training attribute multiple times to createmultiple decision trees.

In step 306, a mean decrease in an accuracy index for each of theattributes associated with the pair of defect types is calculated withthe first ensemble learning classifier. For example, the one or moreprocessors 106 may calculate, with the first ensemble classifier, a meandecrease in the accuracy index for each (or at least some) of theattributes associated with a pair of defect types. For instance, the oneor more processors 106 may calculate a mean decrease in the accuracyindex for each (or at least some) of the attributes associated with afirst defect type and a second defect type (or a second defect type anda third defect type, a first defect type and a third defect type and soon) using the first ensemble learning classifier.

In step 308, the N attributes having the largest mean decrease inaccuracy index are identified. It is noted herein that a mean decreasein an accuracy index for a pair of defect types (e.g., first defecttype/second defect type pair) represents a measure of the attributeswhich are best suited (or at least sufficiently suited) to separate thefirst defect type and the second defect type of the given pair of defecttypes.

In one embodiment, the one or more processors 106 may sort the meandecrease in accuracy index and then identify a selected number (N)having the largest mean decrease in accuracy index. In anotherembodiment, the one or more processors 106 may sort the accuracy indexin a selected fashion (e.g., based on mean decrease, median decrease andetc.) and identify a selected number of the attributes displaying aselected feature (e.g., largest mean decrease, largest median decrease,a selected number from the N attributes having the largest mean ormedian decrease in accuracy). It is noted herein that the presentdisclosure and method 300 are not limited to the statistical analysis ofstep 308, which is provided merely for illustrative purposes. Rather, itis noted that step 308 should be interpreted to extend to anystatistical analysis process used to identify a sub-set of theattributes associated with a decrease in accuracy index.

In step 310, a second ensemble learning classifier is generated, ortrained, using the identified attributes of step 308. In one embodiment,the one or more processors 106 may train a second ensemble learningclassifier using the N attributes having the largest mean decrease inaccuracy index. The second ensemble learning classifier may include anyensemble learning classifier known in the art. For example, as discussedpreviously herein, the second ensemble learning classifier may include,but is not limited to, a random forest classifier or a SVM-basedclassifier.

In step 312, a training error associated with the second generatedensemble learning classifier is determined. In one embodiment, the oneor more processors 106 determine a training error associated with thesecond generated ensemble learning classifier. For example, the one ormore processors 106 may determine an out-of-bag (OOB) training errorassociated with the second generated ensemble learning classifier. Thedetermination of OOB error is generally discussed by Breiman in RandomForests, Machine Learning, Vol. 45, Issue 1, pp. 5-32 (2001), which isincorporated above by reference in the entirety; and Kulkarni et al. inRandom Forest Classifiers: A Survey and Future Research Directions,International Journal of Advanced Computing, Vol. 36, Issue 1, pp.1144-1153 (2013), which is incorporated above by reference in theentirety.

In step 314, a type pair score associated with the pair of defect typesis calculated. In one embodiment, the one or more processors 106 maycalculate a type pair score associated with the pair of defect typesbased on the training error determination of step 312. For example, theone or more processors 106 may calculate a type pair score associatedwith the first defect type and the second defect type (or a seconddefect type and a third defect type, a first defect type and a thirddefect type and so on) based on the training error determination of step312. For example, in the case where OOB error is used to calculate thetraining error, the defect type pair score may be calculated using thefollowing relationship:

Type Pair Score=1−OOB error

In another embodiment, the one or more processors 106 may compare thecalculated type pair score to a predefined threshold. In anotherembodiment, in the event that the type pair score is lower than apredefined threshold (e.g., selected via user input), the one or moreprocessors 106 may provide an indication that the attributes selected instep 308 are insufficient to separate the defect classes of the defectclass pair (e.g., first defect class and second defect class). Further,the one or more processors 106 may report the type pair score to theuser via display 114 of the user interface device 110 or to memory 108of controller 104.

In another embodiment, the controller 104 may monitor the type pairscore throughout a semiconductor device production process. It is notedherein that a reduction in the type pair score may indicate theformation of one or more new defect classes in the given specimen. Inthis regard, monitoring the type pair score provides an indication ofwhether a new defect class has formed during the production process.

Further, in the case where additional defect type pairs require analysisstep 316 may cause the method to repeat steps 304-314 to determine atype pair score for additional defect type pairs. In the case where noadditional defect type pairs require analysis, the method 300 ends.

FIG. 4 is a flow diagram illustrating steps performed in a method 400 ofdetermining a defect data sufficiency score, in accordance with oneembodiment of the present disclosure. It is noted herein that the stepsof method 400 may be implemented all or in part by the system 100. It isfurther recognized, however, that the method 400 is not limited to thesystem 100 in that additional or alternative system-level embodimentsmay carry out all or part of the steps of method 400. It is furthernoted that the embodiments and examples described in the context ofmethods 200 and 300 should be interpreted to extend to method 400 (andvice versa) unless otherwise noted.

In step 402, one or more images 107 are acquired including multipledefect types. In one embodiment, the inspection tool 102 acquires one ormore images 107 and transmits the one or more images 107 to the one ormore controllers 104.

In step 404, a manual classification of the one or more defect data iscarried out. For example, a user may manually classify the defects ofthe defect data contained in the one or more images 107. For instance, auser may manually classify the defects via user interface device 110based on one or more attributes of the one or more defects. In turn, theuser interface device 110 may transmit a signal indicative of the manualclassification of the one or more defects of the specimen 112 to the oneor more processors 106 of controller 104. In another embodiment, thecontroller 104 may receive the manual classification of the defects andstore the result in memory 108. The one or more attributes used to carryout the classification of step 404 include any one or more attributesthat can be derived from a defect inspection or review tool. Forexample, the one or more attributes may include, but are not limited to,image feature amounts, defect coordinates, composition analysis results,manufacture initiation history data, machine QC (Quality Control) data.Further, in some embodiments, the one or more attributes may be obtainedfrom multiple types of defect inspection tools or systems, such as, butnot limited to, an optical or SEM foreign matter inspection machine, apattern inspection machine, a defect review machine, SPM, or anelemental analysis machine. Attributes suitable for classification ofdefects are described in U.S. Pat. No. 7,602,962, issued on Oct. 13,2009, which is incorporated previously herein by reference in theentirety.

In one embodiment, the attributes of the one or more training defectsmay be processed by an automatic classification function, such as, butnot limited to, a real time automatic classification (RT-ADC) toclassify one or more training defects. It is again noted that theutilization of RT-ADC provides for a “rough” automatic classification oftraining defects without sacrificing high processing speeds. Real timeautomatic classification is again described generally in U.S. Pat. No.7,602,962, issued on Oct. 13, 2009, which is incorporated above byreference in the entirety.

In step 406, the defect data is distributed into N groups. For example,following the manual classification of defect data in step 404, the oneor more processors 106 may distribute the defect data into N groups. Forinstance, the number of groups N may be selected via user input. Inanother embodiment, the defect data may be randomly distributed into Ngroups. Further, it is recognized herein that the distribution of defectdata into N groups may be carried out by distributing and storing thedefect data into a set of N folders maintained in memory 108 (or anothermemory).

In step 408, one of the N groups of defect data is identified as testdefect data. In step 410, the remaining N-1 groups of defect data areidentified as training defect data. For example, the one or moreprocessors 106 may identify, or select, one of the groups of distributeddefect data to serve as test data, while the remaining N-1 groups servesa training data. Once the N-1 groups of defect data have been identifiedas training defect data the N-1 groups of data may be combined and usedfor analysis in the following steps.

In steps 412 and 414, for at least a first group of the N groups ofdefect data, a series of classifiers and the corresponding accuracyvalue for each classifier are incrementally generated. It is noted thatthese steps may be repeated for additional groups (e.g., folders) of theN groups via step 418 discussed further herein. In step 412, an i^(th)classifier is generated based on the training data contained in the N-1groups for the case where the training data contains a selectedpercentage (C_(i)) of a first defect type. In this regard, a series ofclassifiers may be incrementally generated for each percentage C_(i) ofthe first defect type. For example, each classifier is generated with anincremented percentage of at least a first defect type contained withinthe training defect data of the N-1 groups of data. For instance, aseries of classifiers may be generated at each 10% increment of firstdefect percentage from 0 to 100%. For example, a first classifier may begenerated at a first defect percentage (C₁) of 10%, a second classifiermay be generated at a second defect percentage (C₂) of 20%, a thirdclassifier may be generated at a third defect percentage (C₃) of 30%, afourth classifier may be generated at a fourth defect percentage (C₄) of40%, a fifth classifier may be generated at a fifth defect percentage(C₅) of 50%, a sixth classifier may be generated at a sixth defectpercentage (C₆) of 60%, a seventh classifier may be generated at aseventh defect percentage (C₇) of 70%, an eighth classifier may begenerated at an eighth defect percentage (C₈) of 80%, a ninth classifiermay be generated at a ninth defect percentage (C₉) of 90% and a tenthclassifier may be generated at a tenth defect percentage (C₁₀) of 100%.It is noted that method 400 is not limited to these increments and thepresent disclosure should be interpreted to extend to any incrementingstandard (e.g., 2%, 5%, 10%, 20% increments and so on). For instance,classifiers may be generated only for the cases where the first defectpercentage is 80%, 90% and 100%.

It is noted herein that the classifiers generated in this step mayinclude any classifier known in the art and is not limited to a ensemblelearning classifier. In one embodiment, one or more of the generatedclassifiers are ensemble learning classifiers (e.g., random forestclassifiers, SVM-based classifiers and the like). In another embodiment,one or more of the generated classifiers are single decision treeclassifiers or multiple decision tree classifiers (e.g., superclassifier).

In step 414, an accuracy value is determined for each of the i^(th)classifiers for the first defect type on the test data. For example, theone or more processors 106 may determine an accuracy value for each ofthe series of classifiers generated in step 412 for at least the firstdefect type. In this regard, the one or more processors may determine anaccuracy value for each of the classifiers by applying each of theclassifiers to the test data not contained in the N-1 groups of thedistributed defect data.

As noted previously herein, steps 412 and 414 may be repeated for eachincrement of the first defect type percentage. Once each of theclassifiers and corresponding accuracy values have been calculated foreach Ci percentage of the first defect type (or additional defect typesin further iterations—see step 422) the method 400 moves to step 416.

In step 416, an accuracy score is generated for the first defect type.In one embodiment, the one or more processors 106 generate an accuracyscore for the first defect type by aggregating the accuracy valuescalculated in step 414 for each Ci percentage of the first defect type.For example, the accuracy score may be generated by the one or moreprocessors 106 by determining a statistical measure of the deviation ofthe accuracy values associated with the Ci percentages. In oneembodiment, the one or more processors 106 may determine the accuracyscore of the first defect type by via the following relationship:

Accuracy Score=1−s*std(accuracy values of C _(i) percentages)

where s represents a scaling factor and std represents the standarddeviation of the accuracy values of the C_(i) percentages of the firstdefect type acquired in step 414.

In step 418, the method 400 may repeat steps 408-416 for each group of Ngroups. In this regard, the method may calculate an accuracy score foreach of the N groups. Once the method 400 has performed the accuracyscore determination for each of the N groups, the method moves to step420.

In step 420, a data sufficiency score for the first defect type iscalculated. For example, the one or more processors 106 may calculate adata sufficiency score by aggregating the accuracy score found in step416 for each of the N groups and then calculating an average accuracyscore for the N groups. In this regard, the data sufficiency score forthe first defect type may take the form:

${{Data}\mspace{14mu} {Sufficiency}\mspace{14mu} {Score}} = {\frac{1}{N}{\sum\limits_{j = 1}^{N}{{Accuracy}\mspace{14mu} {Score}_{j}}}}$

Once the method 400 has performed the data sufficiency score for thefirst defect class, the method 400 may then move to step 422.

In step 422, the method 400 may repeat steps 408-420 for one or moreadditional defect classes. In this regard, the method may calculate adata sufficiency score (step 420) for any number of defect classes. Onceall of the desired defect classes have been scored the method 400 thenends.

All of the methods described herein may include storing results of oneor more steps of the method embodiments in a storage medium. The resultsmay include any of the results described herein and may be stored in anymanner known in the art. The storage medium may include any storagemedium described herein or any other suitable storage medium known inthe art. After the results have been stored, the results can be accessedin the storage medium and used by any of the method or systemembodiments described herein, formatted for display to a user, used byanother software module, method, or system, etc. Furthermore, theresults may be stored “permanently,” “semi-permanently,” temporarily, orfor some period of time. For example, the storage medium may be randomaccess memory (RAM), and the results may not necessarily persistindefinitely in the storage medium.

Those having skill in the art will recognize that the state of the arthas progressed to the point where there is little distinction leftbetween hardware and software implementations of aspects of systems; theuse of hardware or software is generally (but not always, in that incertain contexts the choice between hardware and software can becomesignificant) a design choice representing cost vs. efficiency tradeoffs.Those having skill in the art will appreciate that there are variousvehicles by which processes and/or systems and/or other technologiesdescribed herein can be effected (e.g., hardware, software, and/orfirmware), and that the preferred vehicle will vary with the context inwhich the processes and/or systems and/or other technologies aredeployed. For example, if an implementer determines that speed andaccuracy are paramount, the implementer may opt for a mainly hardwareand/or firmware vehicle; alternatively, if flexibility is paramount, theimplementer may opt for a mainly software implementation; or, yet againalternatively, the implementer may opt for some combination of hardware,software, and/or firmware. Hence, there are several possible vehicles bywhich the processes and/or devices and/or other technologies describedherein may be effected, none of which is inherently superior to theother in that any vehicle to be utilized is a choice dependent upon thecontext in which the vehicle will be deployed and the specific concerns(e.g., speed, flexibility, or predictability) of the implementer, any ofwhich may vary. Those skilled in the art will recognize that opticalaspects of implementations will typically employ optically-orientedhardware, software, and or firmware.

While particular aspects of the present subject matter described hereinhave been shown and described, it will be apparent to those skilled inthe art that, based upon the teachings herein, changes and modificationsmay be made without departing from the subject matter described hereinand its broader aspects and, therefore, the appended claims are toencompass within their scope all such changes and modifications as arewithin the true spirit and scope of the subject matter described herein.

Furthermore, it is to be understood that the invention is defined by theappended claims. It will be understood by those within the art that, ingeneral, terms used herein, and especially in the appended claims (e.g.,bodies of the appended claims) are generally intended as “open” terms(e.g., the term “including” should be interpreted as “including but notlimited to,” the term “having” should be interpreted as “having atleast,” the term “includes” should be interpreted as “includes but isnot limited to,” etc.). It will be further understood by those withinthe art that if a specific number of an introduced claim recitation isintended, such an intent will be explicitly recited in the claim, and inthe absence of such recitation no such intent is present. For example,as an aid to understanding, the following appended claims may containusage of the introductory phrases “at least one” and “one or more” tointroduce claim recitations. However, the use of such phrases should notbe construed to imply that the introduction of a claim recitation by theindefinite articles “a” or “an” limits any particular claim containingsuch introduced claim recitation to inventions containing only one suchrecitation, even when the same claim includes the introductory phrases“one or more” or “at least one” and indefinite articles such as “a” or“an” (e.g., “a” and/or “an” should typically be interpreted to mean “atleast one” or “one or more”); the same holds true for the use ofdefinite articles used to introduce claim recitations. In addition, evenif a specific number of an introduced claim recitation is explicitlyrecited, those skilled in the art will recognize that such recitationshould typically be interpreted to mean at least the recited number(e.g., the bare recitation of “two recitations,” without othermodifiers, typically means at least two recitations, or two or morerecitations). Furthermore, in those instances where a conventionanalogous to “at least one of A, B, and C, etc.” is used, in generalsuch a construction is intended in the sense one having skill in the artwould understand the convention (e.g., “a system having at least one ofA, B, and C” would include but not be limited to systems that have Aalone, B alone, C alone, A and B together, A and C together, B and Ctogether, and/or A, B, and C together, etc.). In those instances where aconvention analogous to “at least one of A, B, or C, etc.” is used, ingeneral such a construction is intended in the sense one having skill inthe art would understand the convention (e.g., “a system having at leastone of A, B, or C” would include but not be limited to systems that haveA alone, B alone, C alone, A and B together, A and C together, B and Ctogether, and/or A, B, and C together, etc.). It will be furtherunderstood by those within the art that virtually any disjunctive wordand/or phrase presenting two or more alternative terms, whether in thedescription, claims, or drawings, should be understood to contemplatethe possibilities of including one of the terms, either of the terms, orboth terms. For example, the phrase “A or B” will be understood toinclude the possibilities of “A” or “B” or “A and B.”

It is believed that the present disclosure and many of its attendantadvantages will be understood by the foregoing description, and it willbe apparent that various changes may be made in the form, constructionand arrangement of the components without departing from the disclosedsubject matter or without sacrificing all of its material advantages.The form described is merely explanatory, and it is the intention of thefollowing claims to encompass and include such changes.

What is claimed:
 1. A method for determining the sufficiency of defectdata for classification comprising: acquiring a set of defect data froma specimen, the defect data including imagery data associated with aplurality of defects including a plurality of defect types; receiving asignal from a user interface device indicative of a manualclassification of the plurality of defects; distributing defect data ofat least the first defect type into N groups of data; identifying agroup of the N groups of data as containing test data; identifying N-1groups of data of the distributed defect data not identified ascontaining test data as containing training data; for at least a firstgroup of the N groups, incrementally generating a series of classifiersbased on the training defect data contained in the N-1 groups of data,wherein each classifier is generated with an incremented percentage ofat least a first defect type contained within the training defect dataof the N-1 groups of data; determining an accuracy value for each of theseries of classifiers for at least the first defect type by applyingeach of the series of classifiers to the test data not contained in theN-1 groups of the distributed defect data; and generating a defect datasufficiency score, for at least the first defect type, based on agenerated accuracy score for at least the first group of N groups and atleast one additional generated accuracy score for at least oneadditional group of the N groups.
 2. The method of claim 1, wherein thegenerating a defect data sufficiency score, for at least the firstdefect type, based on a generated accuracy score for at least the firstgroup of N groups and at least one additional generated accuracy scorefor at least one additional group of the N groups comprises: generatingan accuracy score for at least the first group of N groups based on theaccuracy values determined for each of the series of classifiers for atleast the first defect type; generating an accuracy score for at leastan additional group of N groups based on the accuracy values determinedfor each of a series of additional classifiers for at least the firstdefect type; and generating a defect data sufficiency score, for atleast the first defect type, based on the generated accuracy score forat least the first group of N groups and the at least one additionalgenerated accuracy score for the at least one additional group of the Ngroups.
 3. The method of claim 1, further comprising: generating adefect data sufficiency score, for at least an additional defect type,based on a generated accuracy score for at least the first group of Ngroups and at least one additional generated accuracy score for at leastone additional group of the N groups.
 4. The method of claim 1, whereinthe N-1 groups of training data are aggregated into a single data group.5. The method of claim 1, wherein incrementally generating a series ofclassifiers based on the training defect data contained in the N-1groups comprises: incrementally generating a series of ensemble learningclassifiers based on the training defect data contained in the N-1groups.
 6. The method of claim 5, wherein incrementally generating aseries of classifiers based on the training defect data contained in theN-1 groups comprises: incrementally generating a series of random forestclassifiers based on the training defect data contained in the N-1groups.
 7. The method of claim 1, wherein the distributing defect dataof at least the first defect type into N groups comprises: randomlydistributing defect data of at least the first defect type into Ngroups.
 8. The method of claim 1, wherein the one or more trainingdefects are manually classified with a real-time automatic defectclassification (RT-ADC) scheme applied to the one or more attributes. 9.An apparatus for determining sufficiency of defect data forclassification comprising: an inspection tool, the inspection toolincluding one or more detectors configured to acquire one or more imagesof at least a portion of a specimen; a user interface device; and acontroller, the controller including one or more processorscommunicatively coupled to the one or more detectors of the inspectiontool, wherein the one or more processors are configured to execute a setof program instructions stored in memory, the set of programinstructions configured to cause the one or more processors to: receivethe set of defect data from a specimen, the defect data includingimagery data associated with a plurality of defects including aplurality of defect types; and generate a defect data sufficiency score,for at least a first defect type, based on a generated accuracy scorefor at least a first group of N groups of the defect data and at leastone additional generated accuracy score for at least one additionalgroup of the N groups of defect data.
 10. The apparatus of claim 9,wherein at least one of the series of classifiers comprises: a randomforest classifier.
 11. The apparatus of claim 9, wherein the inspectiontool comprises: an electron beam defect review tool.
 12. The apparatusof claim 9, wherein the inspection tool comprises: a darkfieldinspection tool.
 13. The apparatus of claim 9, wherein the inspectiontool comprises: a brightfield inspection tool.
 14. The apparatus ofclaim 9, wherein the one or more processors are configured to generate adefect data sufficiency score, for at least a first defect type, basedon a generated accuracy score for at least a first group of N groups ofthe defect data and at least one additional generated accuracy score forat least one additional group of the N groups of defect data by:generating an accuracy score for at least a first group of N groupsbased on accuracy values determined for each of a series of classifiersfor at least the first defect type; generating an accuracy score for atleast an additional group of N groups based on accuracy valuesdetermined for each of a series of additional classifiers for at leastthe first defect type; and generating a defect data sufficiency score,for at least the first defect type, based on the generated accuracyscore for at least the first group of N groups and the at least oneadditional generated accuracy score for the at least one additionalgroup of the N groups.
 15. The apparatus of claim 14, wherein the one ormore processors are further configured to: generate a defect datasufficiency score, for at least an additional defect type, based on agenerated accuracy score for at least the first group of N groups and atleast one additional generated accuracy score for at least oneadditional group of the N groups.